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Decision support systems are applications that provide meaningful 
information and help with decision support. To gain the benefits of these 
applications, they need to be used optimally. Thus, the optimal manner to 
utilize relies on several properties, such as usability. This research aims to 
investigate the decision support systems applications (DSS) in terms of 
usability evaluation and propose a usability instrument for DSS in a fog 
computing environment. Recently, complexity is a major problem with most 
software products. Hence, the way to ensure user satisfaction is by measuring 
its usability. The study is based on the research question: What dimensions 
and their elements that utilize to evaluate the usability of DSS applications. A 
systematic literature review (SLR) was conducted to understand and extract 
all the instrument’s dimensions and elements. To confirm the validity of the 
proposed instrument, multi scientific methods were adapted such as face 
validity, a pilot study for testing the goodness of proposed instrument 
consistency, and factor analysis. Furthermore, Bartlett’s test was utilized to 
ensure the reliability of the developed instrument. Eventually, the validated 
instrument was used to evaluate DSS application in a fog computing 
environment. The findings show that the proposed instrument is workable in 
practice. 
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1. INTRODUCTION 


The decision-making process is an inherently human activity that can have worthy impacts [1], [2]. It 
is probably not surprising that researchers have tried to improve the goodness of decisions by utilizing computer 
technologies to increase and extend human capabilities [3]. Decision support systems (DSS) is a form of 
software system that can provide aid to organizations to tackle vital heterogeneous and multi-sources data and 
transform it into useful information. DSS also performs analysis and several data management processes and 
represents one of the main areas of information systems. The context of the study is a decision support system 
in a fog computing environment where the primary actor is a DSS staff involved in providing decision support 
on DSS applications. The DSS staff became very aware of the DSS usability problems and also of a deficiency 
of a DSS usability evaluation process [4]. 
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2. LITERATURE REVIEW 
2.1. Decision support systems 

In recent decades, with the emergence of big data for the abundance of data, it is abundantly clear that 
most decisions are based on data. Data is a backbone for decision support systems. In the same aspect, decision 
support systems target to support decision-makers in making the right decisions at the right time [5],[6]. 


2.2. Software testing in DSS applications 

According to [7], usability testing is to answer whether the system satisfies the user requirement and 
needs. In the same aspect, usability can be defined as the extent to which a software product can be easily 
adopted and used by users to achieve specific goals efficiently and effectively. Besides, Nilson and Cameron 
[8] states that the process of usability assessment of software incorporates three principal goals; i) to what 
extent the user can easily access and use the system's functions, ii) to measure the experience of users 
interacting with this system, and iii) to determine if there are any particular problems using this system [7]. 


2.3. Usability in DSS applications 

DSS usability can be considered a significant factor in DSS application for determining the best of it. 
DSS aims to help and support the massive and heterogeneous data sources and flow of relevant data by 
determining and handling the information into meaningful information [9]. Besides, to gain from the actual 
users about the DSS usability and according to [10], usability testing for DSS applications should be conducted. 
According to the institute of medicine, heightening the usability of DSS applications leads to reducing errors 
and making better decisions [11], [12]. 


3. USABILITY MEASUREMENT 

According to [13], [14], identifying the usability metrics is easier than collecting them. Often, usability 
is measured based on users' satisfaction with a given application. Usability measures are based on the extent to 
which the functional and non-functional requirements are met for the intended application. Accordingly, 
proposing and developing a usability test instrument will also depend on the extent to which the services for 
this application are met for user needs, and of course, according to the nature of the intended application [15]. 
As it is evident that the applications have become customized and each application is designed and developed 
to solve a specific problem or for pre-defined purposes, so the usability test for this application also depends 
on the main purpose of this application. Accordingly, and in the context of this study, the proposed instrument 
for measuring the usability of DSS applications will focus on five important dimensions [1]. Besides, usability 
testing focuses on the actual's users' experience with the intended application, as well as their comments and 
feedback during the use of the such application in their environments [13]. 


4. INSTRUMENT DESIGN AND DEVELOPMENT 

To measure users’ experience towards measuring usability for DSS applications. In this study, the 
questionnaire as an evaluation instrument for DSS applications has been used for its reliability and wide 
adoption [16], [17]. To meet the study objectives, the questionnaire based-instrument is designed to elicit 
comments, suggestions, and feedback regarding DSS usability, and is named (QU-DSS). Since the proposed 
instrument (DU-DSS) has concerned to usability of DSS, the QU-DSS usability dimensions and their items 
has identified through systematic literature review (SLR). Figure 1 depicted the SLR protocol. 


Automated Articles Repetition Deletion Deletion Deletion Adapted 
Search Studies by Title by Reading Abstract by Reading Abstract P 
541 288 203 125 48 
Articles Articles Articles Articles Articles 


Figure 1. SLR protocol 
Indonesian J Elec Eng & Comp Sci, Vol. 30, No. 3, June 2023: 1689-1698 


Indonesian J Elec Eng & Comp Sci ISSN: 2502-4752 O 1691 


According to [18], argue that adapted a systematic approach is a significant in developing evaluation 
instrument. Figure 2 presents the main phases of the evaluation instrument design. As indicated from Figure 2, 
the designing procedure of the QU-DSS start with SLR to identify the instrument dimensions and their items. 
Consequently, the findings of SLR depict that the QU-DSS evaluation instrument have five dimensions named: 
usefulness, simplicity, reliability, flexibility, and decision-support. Several standard instruments such as 
questionnaire for user interface satisfaction (QUIS) by Chin et al. [19], perceived usefulness and ease of use 
(PUEU) by Davis [20], software usability measurement inventory (SUMI) by Kirakowski ef al. [21], and 
practical heuristics for usability evaluation (PHUE) by Chung and Sahari [22] were widely adapted widely 
adapted and they have similar dimensions; therefore, most of the relevant dimensions’ items have adapted and 
utilized and later the first draft of QU-DSS was released. 

As indicated in Table 1, The proposed instrument had formed a sequence of items (questions) with a 
suitable answer for each question. Accordingly, the QU-DSS instrument was utilized to measure the usefulness, 
simplicity, flexibility, reliability, as well as decision support under usability for DSS applications based on a 
fog computing environment. 


SLR Usability Usability Develop of Final Draft of 
Dimensions Goals Instrument Instrument 


Construct Validity 
Statistical 
Procedure Relability Value of Cronbach’s 
Alpha 


Figure 2. Evaluation instrument design (main phases) 


Table 1. The proposed instrument (first draft) 
No QU-DSS dimensions QU-DSS items Author and year 
1 Usefulness (UFL) 7 items QU-DSS-UFL-1 [23], [24] and proposed by the researcher 
QU-DSS-UFL-2 
QU-DSS-UFL-3 
QU-DSS-UFL-4 
QU-DSS-UFL-5 
QU-DSS-UFL-6 
QU-DSS-UFL-7 
2 Simplicity (SMP) 6 items QU-DSS-SMP-1 [25], [26] and proposed by the researcher 
QU-DSS-SMP-2 
QU-DSS-SMP-3 
QU-DSS-SMP-4 
QU-DSS-SMP-5 
QU-DSS-SMP-6 
3 Reliability (REL) 7 items QU-DSS-REL-1 — [24], [27], [28] and proposed by the researcher 
QU-DSS-REL-2 
QU-DSS-REL-3 
QU-DSS-REL-4 
QU-DSS-REL-5 
QU-DSS-REL-6 
QU-DSS-REL-7 
4 decision support (DS) 7 items QU-DSS-DS-1 [9], [29], [30] and proposed by the researcher 
QU-DSS-DS-2 
QU-DSS-DS-3 
QU-DSS-DS-4 
QU-DSS-DS-5 
QU-DSS-DS-6 
QU-DSS-DS-7 
5 Flexibility (FLX) 6 items QU-DSS-FLX-1 [31 ]-[35] and proposed by the researcher 
QU-DSS-FLX-2 
QU-DSS-FLX-3 
QU-DSS-FLX-4 
QU-DSS-FLX-5 
QU-DSS-FLX-6 
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The QU-DSS was piloted to measure its reliability and validity before deliver it to in real environment 
in measuring DSS application based on fog computing environment. Besides, [36] in this study, researchers 
used “somewhat agree” instead of the “neutral” option to force respondents to choose a side. Thus, a 5-point 
likert-type scale has been utilized in the research, and the interpretations of the scales: strongly agree; agree; 
somewhat agree; disagree, and strongly disagree. 


5. PILOT TEST: MEASURING INSTRUMENT GOODNESS 

To evaluate the whole QU-DSS instrument under survey conditions. A pilot study was conducted. By 
pilot testing, the instrument's problems are determined before utilizing the full examination. Pilot testing 
focuses on each question in the proposed instrument and exam the validity. It checks whether the item is 
interpreting and representing the information it’s intended to measure [37], [38]. 


5.1. Subject selection 

In the context of this study, the pilot test conducted with 84 respondents was obtained among the 
college of computer science postgraduate students as well as CS lecturers. The respondent number is enough 
to achieve reliable findings in the statistical test, as supported by [39]-[41]. The total number of sample 
participants was 84. The pilot test data showed 50 females (59%) and 34 males (41%). Most of the participants 
revealed their age to be above 30. 


5.2. QU-DSS validation 

In the context of this study, two well-known methods were adopted to validate QU-DSS instrument, 
the first one is content validity while the second one is interitem consistency analysis. for content validity, face 
validity was utilized as supported by [42]. The justification for conducting face validity to confirm that the 
QU-DSS proposed instrument includes an adequate set of measuring elements of the intended dimension. 
Accordingly, this study employed seven experts face-to-face and through e-mail to review the QU-DSS items 
in terms of validity of the content. The review finding of the experts indicates that some QU-DSS items were 
not suitable sufficiently to use and did not meet well with the intended QU-DSS dimensions. Consequently, 
QU-DSS's first draft was modified in terms of repositioning, rewording, and sometimes discarding some 
irrelevant items. To confirm consistency, the reliability test was conducted as a second method for QU-DSS 
instrument. Thus, consistency is measured by the reliability test. The Cronbach’s coefficient alpha (a ) was 
counted [43], [44]. This study conducted a test and considered (a > 0.6) to be significant, as supported by [45]. 
Table 2 depict the findings of the reliability test of the QU-DSS measurement items, the findings indicate that 
QU-DSS instrument are consistent and significant, and can be adapted to utilize for measuring the usability 
and decision support for DSS application in for computing environment. 


Table 2. Finding of measuring reliability 
No QU-DSS dimensions __No. of items __Cronbach’s alpha 


1 QU-DSS-UFL 7 0.783 
2 QU-DSS-SMP 6 0.725 
3 QU-DSS-REL 7 0.714 
4 QU-DSS-DS 7 0.791 
5 QU-DSS-FLX 6 0.708 


5.3. Factor analysis process 

To confirm the degree of significance of QU-DSS items, factor analysis was conducted. The accepting 
items are selected based on Kaiser-Meyer-Olkin (KMO) and Bartlett’s test, and the value of factor loading 
[44], [46], [47]. Consequently, to organize the data for factor loading, KMO test was conducted. As indicated 
in Table 3, the obtained findings of KMO test have accepted based on the condition of KMO test > 0.50. 


Table 3. The KMO overall findings 
No QU-DSS dimensions _ No. of items _ KMO _ The value of Bartlett’s test 


1 QU-DSS-UFL 7 0.783 0.000 
2 QU-DSS-SMP 6 0.725 0.000 
3 QU-DSS-REL 7 0.714 0.000 
4 QU-DSS-DS 7 0.791 0.000 
5 QU-DSS-FLX 6 0.708 0.000 
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As indicated from Table 3 the Bartlett’s test of sphericity value is 0.000 for all dimensions, this 
indicates that the second condition is met (p < 0.05). Thence, this motivates that the data are willing and able 
for factor loading analysis test. Table 4 detailed the factor loading test. As can be seen in Table 4, all items in 
QU-DSS are usable and represent the respective dimensions except the three items marked with an asterisk (*) 
are excluded from the test and whose factor loading values are less than 0.50 as supported by [48]. 


Table 4. The factor loading overall findings 


No QU-DSS dimensions QU-DSS items Factor loading value 
1 Usefulness (ufl) 7 items QU-DSS-UFL-1 0.587 
QU-DSS-UFL-2 0.661 
QU-DSS-UFL-3 0.625 
QU-DSS-UFL-4 0.554 
QU-DSS-UFL-5 0.362* 
QU-DSS-UFL-6 0.511 
QU-DSS-UFL-7 0.531 
2 Simplicity (smp) 6 items QU-DSS-SMP-1 0.686 
QU-DSS-SMP-2 0.534 
QU-DSS-SMP-3 0.541 
QU-DSS-SMP-4 0.661 
QU-DSS-SMP-5 0.672 
QU-DSS-SMP-6 0.572 
3 Reliability (REL) 7 items QU-DSS-REL-1 0.645 
QU-DSS-REL-2 0.387* 
QU-DSS-REL-3 0.615 
QU-DSS-REL-4 0.609 
QU-DSS-REL-5 0.694 
QU-DSS-REL-6 0.621 
QU-DSS-REL-7 0.566 
4 Decision support (DS) 7 items | QU-DSS-DS-1 0.551 
QU-DSS-DS-2 0.686 
QU-DSS-DS-3 0.538 
QU-DSS-DS-4 0.546 
QU-DSS-DS-5 0.665 
QU-DSS-DS-6 0.673 
QU-DSS-DS-7 0.578 
5 Flexibility (FLX) 6 items QU-DSS-FLX-1 0.689 
QU-DSS-FLX-2 0.534 
QU-DSS-FLX-3 0.545 
QU-DSS-FLX-4 0.662 
QU-DSS-FLX-5 0.373* 
QU-DSS-FLX-6 0.571 


6. USE QU-DSS FOR USABILITY MEASUREMENT 

After confirming the validity of the developed tool for use, it was used to measure the usability of a 
clinical decision support system developed previously by the researcher. The actual users of the system in three 
government hospitals (27 clinicians) as well as the sixth stage students belonging to the faculty of medicine 
(73) total (100), were asked to evaluate the developed system in terms of usability. Morevere, to sum up or to 
describe the characteristics of Selected data set or samples, the descriptive statistics was conducted (the mean 
(M) and standard deviation (STD) were calculated). Figures 3 to 8 and Tables 5 to 9 respectively show the 
results obtained for usefulness, simplicity, reliability, decision support, and flexibility through the evaluation 
process. 


Table 5. Usefulness dimension findings 
1 2: 3 4 5 


No Usefulness items STD M 
Frequency and percentage (%) 

1 QU-DSS-UFL1 1.023 4.541 0.0 0.0 3 80 17 
2 QU-DSS-UFL2 1.045 4.672 0.0 0.0 4 78 18 
3 QU-DSS-UFL3 1.101 4.826 0.0 0.0 2 80 18 
4 QU-DSS-UFL4 1.087 4.911 0.0 0.0 3 80 17 
5 QU-DSS-UFLS 1.102 4.541 0.0 0.0 4 78 18 
6 QU-DSS-UFL6 1.031 4.441 0.0 0.0 2 80 18 

Average 3 80 17 


Note: 1, 2, 3, 4, and 5 means :strongly disagree, disagree, somewhat agree, agree, strongly agree 
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Figure 3. Usefulness dimension findings 


In terms of usefulness, the findings details are illustrated in Table 5 and Figure 3. The majority of 
participants are either strongly agree or agree and few of them are somewhat agree. Thus, the DSS application 
that measured by QU-DSS is useful. 


Table 6. Simplicity dimension findings 
1 2 3 4 5 


No Simplicity items STD M Frequency and pacentaee (0) 
1 QU-DSS-SMP1 1.103 4.740 0.0 0.0 7 77 16 
2 QU-DSS-SMP2 1.145 4.274 0.0 0.0 5 74 18 
3 QU-DSS-SMP3 1.055 4.427 0.0 0.0 7 76 17 
4 QU-DSS-SMP4 1.387 4.713 0.0 0.0 7 77 16 
5 QU-DSS-SMPS5 1.215 4.642 0.0 0.0 5 74 18 
6 QU-DSS-SMP6 1.065 4.148 0.0 0.0 7 76 17 
Average 7 71 16 


Note: 1, 2, 3, 4, and 5 means :strongly disagree, disagree, somewhat agree, agree, strongly agree 
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Figure 4. Simplicity dimension findings 


In terms of simplicity, the findings details are illustrated in Table 5 and Figure 4. The majority of 
participants are either strongly agree or agree and few of them are somewhat agree. Thus, the DSS application 
that measured by QU-DSS has the characteristic of simplicity. 


Table 7. Reliability dimension findings 
1 2 3 4 5 


No Reliability items STD M 
Frequency and percentage (%) 
1 QU-DSS-REL1 1.103 4.740 0.0 0.0 15 50 35 
2 QU-DSS-REL2 1.105 4.274 0.0 0.0 14 51 35 
3 QU-DSS-REL3 1.005 4.427 0.0 0.0 16 49 35 
4 QU-DSS-REL4 1.307 4.713 0.0 0.0 15 50 35 
5 QU-DSS-REL5 1.205 4.642 0.0 0.0 14 51 35 
6 QU-DSS-REL6 1.005 4.148 0.0 0.0 16 49 35 


Average 15 50 35 
Note: 1, 2, 3, 4, and 5 means :strongly disagree, disagree, somewhat agree, agree, strongly agree 
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Figure 5. Reliability dimension findings 


In terms of reliability, the findings details are illustrated in Table 6 and Figure 5. The majority of 
participants are either strongly agree or agree and few of them are somewhat agree. Thus, the DSS application 
that measured by QU-DSS confirm are reliable. 


Table 8. Decision-support dimension findings 


sis : 1 2 3 4 5 
No Decision-support items STD M Frequency and percentage (%) 

1 QU-DSS-DS1 1.001 4.942 0.0 0.0 10 65 25 
2 QU-DSS-DS2 1.504 4.715 0.0 0.0 9 67 24 
3 QU-DSS-DS3 1.115 4.827 0.0 0.0 11 66 23 
4 QU-DSS-DS4 1.117 4.419 0.0 0.0 10 65 25 
5 QU-DSS-DS5 1.212 4.547 0.0 0.0 9 67 24 
6 QU-DSS-DS6 1.106 4.348 0.0 0.0 11 66 23 
Average 10 65 25 


Note: 1, 2, 3, 4, and 5 means :strongly disagree, disagree, somewhat agree, agree, strongly agree 
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Figure 6. Decision-support dimension findings 


In terms of decision-support, the findings details are illustrated in Table 7 and Figure 6. The majority 
of participants are either strongly agree or agree and few of them are somewhat agree. Thus, the DSS 
application that measured by QU-DSS can support decision-making process. 


Table 9. Flexibility dimension findings 
1 2 3 4 5 


No Flexibility items STD M 
Frequency and percentage (%) 
1 QU-DSS-FLX1 1.452 4.142 0.0 0.0 11 70 19 
2 QU-DSS-FLX2 1.511 4.275 0.0 0.0 10 72 18 
3 QU-DSS-FLX3 1.712 4.327 0.0 0.0 12 71 17 
4 QU-DSS-FLX4 1.602 4.479 0.0 0.0 11 70 19 
5 QU-DSS-FLX5 1.801 4.547 0.0 0.0 10 72 18 
6 QU-DSS-FLX6 1.105 4.168 0.0 0.0 12 71 17 


Average a 70 19 
Note: 1, 2, 3, 4, and 5 means :strongly disagree, disagree, somewhat agree, agree, strongly agree 
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Figure 7. Flexibility dimension findings 


In terms of flexibility, the findings details are illustrated in Table 8 and Figure 7. The majority of 
participants are either strongly agree or agree and few of them are somewhat agree. Thus, the DSS application 
that measured by QU-DSS are flexible. In line with the above situation, the findings confirm that QU-DSS are 
usable in terms of flexibility, reliability, simplicity, usefulness, and decision-support. Figure 8 illustrate the 
overall usability test findings. 


Usability test findings 
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Figure 8. The overall usability test findings 


7. CONCLUSION 

Decision support systems are very important systems in most organizations, accordingly designing 
high-quality systems in terms of ease of use and decision support is required. The design and development of 
usability assessment tools is one of the necessities that must be available for all types of applications, especially 
decision support systems. Whenever the developed tool is reliable and designed in an organized and systematic 
way, the results of the evaluation are reliable and reliable. 

In this paper, an assessment tool for decision support systems was designed and developed to measure 
the ease of use of decision support systems. This tool consists of five dimensions (usefulness), each dimension 
has own items. After completing the development of the QU-DSS, it was used to evaluate the clinical decision 
support system previously developed by the researcher. The obtained results showed that the QU-DSS is valid 
and usable in evaluating decision support systems. 
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